SUMMARY:

The South Africa PEPFAR program works with over 100 prime partners, who in turn work with over 300 sub

-partners and 350 service delivery sites, to implement HIV and AIDS activities across South Africa. This

immense level of effort poses a significant challenge to the USG in efficiently monitoring and evaluating

programs (mainly because there is no single source from which to obtain PEPFAR data) and in building

monitoring and evaluation (M&E) capacity among partners. Khulisa helps to address these challenges

through a web-based data warehouse (DW) and through on-going independent Data Quality Assessments

(DQA) of PEPFAR partners' data management systems. Both the DW and DQA activities prioritize M&E

capacity building among PEPFAR/South Africa partners.

This project addresses the emphasis areas of Health Management Information Systems, monitoring,

evaluation and reporting, as well as USG database and reporting systems. The main target populations are

the USG and PEPFAR prime partners, sub-partners, and sites in all nine provinces.

BACKGROUND:

Khulisa Management Services is a South African-based consulting firm offering quality management and

technical services to development projects throughout Africa. With PEPFAR funding, Khulisa has

implemented both the DW and DQA activities since FY 2005. With FY 2006 funding, Khulisa conducted

DQAs of 26 PEPFAR partners and provided data quality training for USG staff and partners. This exercise

provided invaluable feedback on risks to data quality regarding reported PEPFAR data, and also sought to

build M&E capacity and improve data management systems (DMS) among PEPFAR partners. These

DQAs were more collaborative than traditional audits, allowing partners to receive advice on how to improve

practices. The proposed DQAs will continue to build partners' understanding and capacity in M&E systems,

as well as improve the overall quality of data they report.

Since October 2004, Khulisa has provided web-based data warehousing services to PEPFAR through a sub

-grant through John Snow Inc. (JSI) funded through the MEASURE Evaluation project. The DW has

transformed a paper-based COP planning and reporting system to a more efficient web-based system with

data integrity. The DW has undergone continuous revisions to address the changing needs of PEPFAR.

The proposed project activities will further support the DW and develop a sustainable and replicable system.

ACTIVITIES AND EXPECTED RESULTS:

ACTIVITY 1 - Data Quality Assessments (DQAs)

The DQAs are designed as a three-phased approach, using standardized tools based on USAID and other

internationally accepted standards. At each phase, the risks to data quality are identified prompting a

dialogue between the assessor and the partner about how to improve systems, resolve problems, and

resolve data quality risks. The findings of each phase, with associated recommendations, are reported in

detail to both the USG and the partner. In addition, the USG receives a summary report for each phase. A

plan for technical assistance is developed between the partner and the USG.

Phase 1: Phase 1 assessments are conducted with a new group of partners as identified by the USG Task

Force. In this phase, the partner's Data Management System (DMS) and associated processes and

procedures are examined through a self-evaluation, followed by a review of the DMS by the Khulisa

assessor. The main objective is to prepare the partner for Phase 2 and familiarize them with the DQA

process.

Phase 2: Phase 2 involves validation and verification of reported data. The assessor uses two selected

indicators (from source) and tracks it through the partner's DMS to evaluate the reported data for validity,

reliability, timeliness, precision and integrity. In the process of conducting Phase 2 the assessor derives

scores for several dimensions of data quality; these scores are interpreted in the context of data quality

risks. Any identified risks are reported to the partner and the USG with recommendations for corrective

action. Partners with high risk scores are issued compliance notes indicating data management and quality

practices that could be improved in specific ways. The compliance notes also provide recommendations for

resolving practices that contribute to compromised data quality of reported results.

Phase 3: Phase 3 is the follow-up visit which is only done with those partners who received a compliance

note based on a high risk score in Phase 2. The assessor re-examines the data quality issues found during

Phase 2 and assesses whether the corrective action taken by the partners reduces the risks that were

outlined. When the assessor and partner achieve consensus on the corrective action, the compliance note

is considered closed. This final visit also serves as an additional opportunity for the partner to receive

technical assistance from the assessor on data quality practices.

ACTIVITY 2: Data Warehouse (DW)

The DW project is an ambitious and unique activity, and has proven to be a useful tool for PEPFAR

reporting and planning. During the last two years, Khulisa built a web-based DW that is password-protected,

through which implementing partners can electronically submit both narrative and quantitative information

on progress towards their expected results as well as their plan for the forthcoming fiscal year. The DW

also allows the USG Task Force to verify submitted data, make adjustments for partner double counts, and

to maintain an audit trail by tracking changes made to data.

Over the last three years, substantial progress has been made in developing a PEPFAR reporting system.

Multiple tests were performed on the system, which brought about numerous adjustments to improve

efficiency and effectiveness. Feedback has been positive so far and USG staff and partners have now

become more "fluent" in using the system. Last year, in addition to the reporting side of the DW, a planning

side has been added to electronically capture information for the COP, enabling the USG to better manage

the large amount of COP data through version control. An online "track changes" function has been added

this year to assist activity managers with their final COP edits.

Activity Narrative:

Currently, the DW captures progress reports (quarterly, annual and semi-annual) and COP data; provides

tools for managing budgets and targets through online, editable grids; provides a tool for the removal of

double counting; tracks data changes through audit trails; and extracts indicator data, sub-partner and site

information.

In FY 2008, Khulisa will continue to maintain and host the DW, with a focus on expanding features for better

use and analysis of program data at the partner level. Specifically, the project will:

-- Continue improving the currently active functions for even greater ease of use by partners and USG staff.

-- Further extend the extraction and reporting capacity for indicator data, sub-partner and site information,

status information and trend data. The extensions will focus on online graphical representation of data

including maps. Manually-produced maps are already a significant aspect of data use and their availability

online, in real time will improve the usage of this data.

-- Further improve partner-level data usage and data quality through site-level data capture for partners

(other than the treatment partners who currently do so). Site-level capture will be started for partners who

request it, starting with Orphans and Vulnerable Children partners and likely followed by Counseling and

Testing partners.

EXPECTED RESULTS:

These two activities will allow the USG Task Force to make better, data-driven programming and planning

decisions at the macro level, as well as assist partners develop and utilize more effective M&E systems.

The sustainable impact of the system will be the partner's ability to make better programming and planning

decisions for their own programs based on accurate and reliable data.

This activity will assist the entire PEPFAR program achieve its goals through effective M&E of partner

achievements in meeting South Africa's portion of the 2-7-10 goals.